It is well understood that consumers of streaming data will tolerate breaks or gaps in video much better than audio. A consumer may hardly notice a temporary drop from 30 frames per second (fps) to 15 fps (a loss of 50% of the video frames), as long as audio continues to play. Consumers may even tolerate very short drops to 5 fps if the audio remains continuous. However, if the audio cuts out, even for very brief periods of time, the results can be very disruptive. Depending on frequency and duration, loss of audio may result in “pops” or other jarring artifacts, failure to hear critical dialogue, etc.
Many video encodings and file formats allow for variable frame rates. Given the ubiquity of multitasking computing environments, it is fairly standard for three dimensional (3D)-intensive computer programs (e.g., modern video games, 3D modeling and rendering software, etc.) to make efficient use of available hardware by rendering “as fast as possible” while still preserving computing resources to remain responsive to user input. The specific rate might be different from machine to machine or minute to minute on the same machine. For example, the program may render at one frame rate when the underlying hardware is relatively idle, but reduce that frame rate, should computing resources become more scarce (e.g., during background ray-tracing).
However, streaming data over a network connection is not quite as simple. It requires the server to know not only the display capabilities of the remote client, as well as the network bandwidth available for transmission, but also to be able to observe and respond to environmental changes. Historically, attempts to address this issue required a client to manually select among several different “qualities” of encodings of the same content. Where latency was high or throughput low, clients tried to “guess” at how much advance data needed to be temporarily stored or “buffered” to ensure uninterrupted playback of the entire stream. Estimates that were inadequate resulted in playback that would “hang” before continuing. Only recently have we seen crude automated mechanisms to vary frame rate, bit rate, or quality in response to changes in client capabilities during playback. However, “skipping”, “lagging”, or “pausing” audio and video remains an issue where servers cannot respond fast enough to unpredicted environmental changes.
The GXF container file format allows for the storage of audio data immediately preceding the video frame data with which it is associated. However, the storage order is specific to that file format.
While various decoders allow for discarding out-of-band (i.e., “late”) audio and video data, to date, no file format or transmission technology exists that allows for the receipt and buffering of audio data well in advance of the video data with which it is associated to insure that audio data will be available, even where video temporarily may not be. In contexts where raw video and audio data are transmitted (e.g., without metadata like timestamps or sequence numbers), if one were to send audio data well before video data, it would result in synchronization errors, which become noticeable where audio precedes or “leads” video by as little as 25-35 ms.
Tooyah, Inc. runs a system that comprises a web browser on a server accessible via a network. The web browser has a screen display and emits audio. This screen display and audio is converted into a stream of audio and video data that is sent over the network to a client. The client comprises software that can receive, decode, and display the stream as it is received. This is somewhat analogous to a screen-sharing application.
In the Tooyah system, the user interacts with the web browser via a remote control device such as an infrared remote control typically used for controlling televisions, or a network-connected device such as an iPhone or iPad. When the user presses a button on the remote control, a button event is transmitted to the server. The web browser receives the button event and updates its display and may also emit audio. As described above, the display updates and new audio are converted to a stream that is viewed and heard by the user. The time between the user pressing the button and the resulting stream data being perceived by the user must be as small as possible to appear responsive. That time should typically be less than 250 ms.
Because that time is so short, the client has very little opportunity to buffer incoming stream data. If audio data is included in the stream, then the client will have no more than 250 ms of encoded audio buffered. When transmitted over the network, the stream is comprised of data packets. If the network used to transmit the stream is the internet, then data packets can be dropped or delayed. As described more generally above, a delay of more than 250 ms in this case would cause the audio decoder to underflow, resulting in artifacts such as audio outages or pops.
What is needed is a technology for allowing streaming audio to be transmitted in advance of streaming video in a data stream without synchronization errors, in order to minimize audio discontinuities, even where video transmission may become degraded.